fix(resmgr): a host being deauthorised has not gone away - #47
Merged
Conversation
added 2 commits
August 18, 2026 17:48
resmgr answers the per-host endpoints with 404 for minutes after a host's last
role is removed, while GET /resmgr/v2/hosts keeps reporting the host, its roles
and its hostconfig_id throughout. Three reads believed that 404 and removed
resources that still existed:
pcd_host_config_assignment -> next apply: 409 HostToHostconfigConflict
pcd_host_cluster_role -> next apply: 403 HostInAuthState
pcd_host_role -> same
Worse than the failed apply, the assignment leaving state is what let a host
config be deleted as unused, which strands the host for good — the case the
delete guard now refuses outright.
So a 404 is no longer taken at face value: hostRecord falls back to the host
list, which is the only view that stays truthful through the window, and a list
that cannot be read is an error rather than an absence. Nothing leaves state on
an unverified 404. The extra request happens on the 404 path alone, so an
ordinary read still costs one call.
Waiting the window out belongs in the writes that must not race it, not in a
Read: a refresh must not block for minutes.
Also bumps the indirect google.golang.org/grpc to 1.82.1 (xDS RBAC and HTTP/2
advisories). The provider's gRPC server only serves the local Terraform CLI over
a private channel and does not use xDS, so exposure was minimal.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes, both patch-level, released as v0.1.9.
A host being deauthorised has not gone away
resmgr answers the per-host endpoints with
404for minutes after a host's last role isremoved, while
GET /resmgr/v2/hostskeeps reporting that host — its roles and itshostconfig_id— throughout. The host has not gone anywhere.pcd_host_config_assignment,pcd_host_cluster_roleandpcd_host_rolebelieved that404and removed live resources from state. The next apply then failed against the reality those
resources never left. Both downstream symptoms were observed on the lab across several runs:
409 HostToHostconfigConflictre-creating an assignment403 HostInAuthStatere-adding a rolehostRecordnow cross-checks a404against the host list, and only a host absent from bothleaves state. An unreadable list is an error, never an absence — dropping a live resource on an
unverified
404is the whole thing being prevented. An ordinary read still costs one request;the extra call happens only on the
404path.Applied to all three read paths. If resmgr ever stops 404ing a host it still lists, this
degrades to the fast path and can be retired.
gRPC 1.82.0 -> 1.82.1
Clears the open HIGH dependabot alert (gRPC-Go xDS RBAC and HTTP/2 vulnerabilities). Indirect
dependency; the provider's gRPC server only ever serves the local Terraform CLI over a private
channel and does not use xDS, so real exposure was minimal.
Testing
Static gate green: build, vet, gofmt,
golangci-lint0 issues, all 11 packages' unit tests.New unit tests cover the window from both sides (
404+ still listed = present;404+ absentfrom the list = gone), the fail-closed path when the list cannot be read, and that the ordinary
path stays at one request.
On the evidence, stated plainly: the defect is evidenced from live lab runs (the two
errors above are in recorded logs). The fix ships on unit tests, not on a fresh live
reproduction — reproducing the window requires a deauth, which reliably triggers two open PCD
product bugs and left the lab needing a full rebuild. That trade was not worth another rebuild.
An adversarial audit of this diff raised 12 findings; 11 did not survive verification. The one
that did was a coverage gap: nothing pinned the guard that keeps a per-host failure which is
not a
404(a 500, an expired token) from being reported as an absence. The shipped code wascorrect — all three callers check
errbeforeknown— but the guard could be deleted with thesuite staying green. Mutation-tested and now covered.